System, method and computer program for guided image capturing of a meal

ABSTRACT

A system including circuitry configured to process image data of a meal to obtain information on the contents of the meal; generate, based on the obtained information, a query with guidance to change image capture settings: and guide a user to pick up at least a part of the meal.

CROSS-REFERENCE TO RELATED APPLICATIONS

This present application is a continuation of U.S. application Ser. No.16/485,452, filed Aug. 13, 2019, which is based on PCT filingPCT/EP2018/054927, filed Feb. 28, 2018, which claims priority to EP17159719.8, filed Mar. 7, 2017, the entire contents of each areincorporated herein by reference.

FIELD OF DISCLOSURE

The present disclosure relates to the field of computer vision, inparticular to systems, methods and computer programs for identificationof food.

BACKGROUND

In computer vision, mathematical techniques are used to detect thepresence of and recognize various items that are depicted in digitalimages. Computer vision tasks include methods for acquiring, processing,analyzing and understanding digital images, and in general, deal withthe extraction of high-dimensional data from the real world in order toproduce numerical or symbolic information, e.g., in the forms ofdecisions. Localized portions of an image, on which specific types ofcomputations are performed to produce visual features, may be used toanalyze and classify objects depicted in the image. Low-level features,such as interest points and edges, edge distributions, colordistributions, shapes and shape distributions, may be computed fromportions of an image and used to detect items that are depicted in theimage. Machine learning algorithms can be used for feature recognition.

Accurate identification of food being consumed is an important task tofor example people who suffer from food-born allergies, who participatein weight-loss programs, and who just enjoy eating and trying new foods.

SUMMARY

According to a first aspect, a system is provided including circuitryconfigured to process multispectral image data of a meal to obtaininformation on the contents of the meal; and generate, based on theobtained information, a query with guidance to change image capturesettings.

According to a second aspect, a method is provided including processingmultispectral image data of a meal to obtain information on the contentsof the meal; and generating, based on the obtained information, a querywith guidance to change image capture settings.

According to a third aspect, a computer program is provided includinginstructions, the instructions when executed on a processor causing theprocessor to perform: processing multispectral image data of a meal toobtain information on the contents of the meal; and generating, based onthe obtained information, a query with guidance to change image capturesettings.

Further aspects of the disclosure are set forth in the dependent claims,the following description and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present disclosure are explained by way of examplewith respect to the accompanying drawings, in which:

FIG. 1 schematically describes a system for recipe generation;

FIG. 2 describes an example of a recipe generation process;

FIG. 3 shows an embodiment of a recipe generation process associatedwith user feedback;

FIG. 4 shows a specific embodiment of a recipe generation processassociated with user feedback;

FIG. 5 a shows an embodiment of a user query on a display of a mobilephone;

FIG. 5 b shows an alternative embodiment of a user query on a display ofa mobile phone;

FIG. 6 shows a further specific embodiment of a recipe generationprocess associated with user feedback;

FIG. 7 a shows an embodiment of a user query on a display of a mobilephone;

FIG. 7 b shows an alternative embodiment of a user query on a display ofa mobile phone;

FIG. 8 shows a further specific embodiment of a recipe generationprocess associated with user feedback;

FIG. 9 shows an embodiment of a user query on a display of a mobilephone;

FIG. 10 shows a further specific embodiment of a recipe generationprocess associated with user feedback;

FIG. 11 shows an embodiment of a user query on a display of a mobilephone;

FIGS. 12 a, b, c show an example of a generated recipe;

FIG. 13 schematically describes an embodiment of a system for recipegeneration via Internet;

FIG. 14 schematically describes an embodiment of a system for recipegeneration via Internet using feedback;

FIG. 15 shows an embodiment of a method for recipe generation using adifference spectrum;

FIG. 16 shows another embodiment of a method for recipe generation usinga difference spectrum;

FIG. 17 a illustrates a first measurement for obtaining a reflectancespectrum, wherein the light source of the mobile reflectometer isswitched off;

FIG. 17 b illustrates a second measurement for obtaining a reflectancespectrum, wherein the light source of the mobile reflectometer isswitched on; and

FIG. 18 illustrates a coordinate transformation.

DETAILED DESCRIPTION OF EMBODIMENTS

The embodiments described below provide a system including circuitryconfigured to process multispectral image data of a meal to obtaininformation on the contents of the meal; and generate, based on theobtained information, a query with guidance to change image capturesettings.

The system may for example include mobile devices, smartphones, tablets,smartwatches, glasses or other kinds of wearable devices. The system mayalso include any such devices in cooperation with a server or cloudprocessing and/or storage system. Still further, the system may alsoinclude a client device and a remote device, e.g. a server device, thatare connected via a network such as the Internet or a LAN.

The circuitry may for example be implemented by a processor, e.g. acentral processing unit (CPU) or the like. The processor may be locatedon a mobile device, a remote workstation or a cloud server. Thecircuitry may also be distributed circuitry that is distributed over amobile device, a remote workstation, and/or a cloud server.

A multispectral image may be obtained by a multispectral imaging devicesuch as a multispectral camera. Spectral imaging may be a combination ofimaging and spectroscopy, where a spectrum is collected at everylocation of an image plane. Multispectral imaging may compriseretrieving for each pixel multiple measurements, each measurementrelating to a specific frequency in the frequency spectrum.Multispectral imaging is not restricted to visible light, but works alsoin ultraviolet and in infrared. A multispectral camera may for examplecapture measurements in the visible color channel from 400-700 nm and anear infrared (NIR) channel from 750-900+nm. Multispectral imaging mayalso include hyperspectral imaging.

Analyzing multispectral image data of a meal to obtain information onthe contents of the meal may include executing one or more featuredetection algorithms, including machine learning algorithms, semanticreasoning techniques, similarity algorithms, and/or other technologiesto, among other things, in an automated feature detection, recognize anddescribe one or more food items that are depicted in a digital image.Some examples of feature detection algorithms that may be used foranalyzing multispectral image data include a histogram of orientedgradients (HoG), an edge orientation histogram, a scale-invariantfeature transform descriptor (SIFT), and a shape context technique.

Image capture settings may be anything that has influence on whatinformation a sensor arrangement such as a multispectral camera iscapturing. Image capture settings may for example include a positionand/or an orientation of a camera, e.g. angles (shooting angle) of acamera such as roll, yaw and pitch. Image capture settings may alsoinclude the distance between a meal and a camera. Still further, imagecapture settings may also include aspects such as separating aningredient of a meal from the other ingredients, or placing a specificingredient of a meal closer to a camera.

Generating a query with guidance for asking a user to change imagecapture settings may for example be implemented by a user interface. Theuser interface may be associated with visual feedback on a mobile deviceor may involve voice feedback via a mobile device.

The circuitry may be configured to generate the query with guidanceaccording to insufficient information on the contents of the meal. Forexample, at each step, feedback can be received from a user to achievemore precise information.

The circuitry may be configured to guide the user to change the attitudeof a camera (e.g. a shooting angle of a camera) to point to otheringredients of the meal.

The circuitry may also be configured to guide the user to pick up atleast a part of the meal. This may allow to clearly show the part of themeal to a camera.

The circuitry may be also configured to guide the user to cut the mealinto parts and show a surface profile of the meal towards a camera.

The circuitry may be also configured to guide the user to move a cameraand to see a particular object in the meal close up.

The circuitry may be configured to generate a recipe of the meal basedon the obtained information on the contents of the meal. This may forexample be done by comparing and matching the meal contents of anidentified meal with those of meals on an existing recipe database, ormore sophisticatedly by full automatic recipe generation process.

The recipe may for example include ingredients information, nutritioninformation, and/or allergen information, as well as cookinginstructions.

The circuitry may for example be configured to calculate calories andrecommend changes to the generated recipe based on the user's health ordiet plan.

The circuitry may be configured to change the recipe generation processbased on feedback, This may allow to improve the precision of recipegeneration based on feedback received by the user.

The system may include a sensor arrangement configured to collectmultispectral image data of a meal. This sensor arrangement may forexample be located on a mobile device such as smart phone, tablet, orwearable devices.

The sensor arrangement may be configured to provide depth information.The sensor arrangement may for example apply stereoscopic imaging,Time-of-Flight imaging (ToF) or other techniques for providing depthinformation. Accordingly, the circuitry may be configured to use depthinformation for volume analysis of a whole meal or each ingredient ofthe meal to determine the quantity of the ingredients for recipes. Depthinformation may also be helpful to computer vision techniques such ashistogram of oriented gradients (HoG), edge orientation histogram,scale-invariant feature transform descriptor (SIFT), and shape contextto determined shapes.

The sensor arrangement may be configured to provide mass spectrographyinformation. By utilizing mass spectroscopy data, the circuitry mayachieve precise content determination. This may allow identifyingvarious kinds of compounds including sugars, salts, oil, andbiomolecules such as proteins. Recent developments in mass spectroscopyhave shown that mass spectroscopy can be integrated into a compactequipment such as mobile phones. For example, the aero-thermo-dynamicmass analysis (AMA) described by Kota Shiba & Genki Yoshikawa inScientific Reports 6, article number 28849 on nature.com can beintegrated into various analytical devices, production lines, andconsumer mobile platforms. Accordingly, such technology can bebeneficially used for food recognition and recipe generation in thecontext of smart phones.

The sensor arrangement may be configured to provide visible images,infrared images, and/or spectral data.

The circuitry may be configured to employ ingredient segmentation on amultispectral image by distinguishing the difference of spectrumproperties of ingredients.

The circuitry may be configured to identify ingredients by analyzingspectrum data.

The circuitry may be configured to use conventional image data (standardRGB image data) for course identification such as the meal nameidentification. This may help improving the efficiency of processing bynarrowing down the candidate meal contents. Such conventional image datamay for example be compared with reference image data stored in areference image data base.

From meal content determination to recipe generation, the method maysimply identify the name of a meal in front of a camera by utilizingmachine learning and refer to existing recipe database based onidentified meal name to extract the recipe of identified meal.

All above described aspects may also be realized as a method or computerprogram. The method, respectively the computer program may collectinformation of a prepared meal based on multispectral information, depthinformation and/or mass spectrography information using camera inaddition to conventional visible images. The method identifies a meal,analyzes meal content, measures volume of ingredients, and generates arecipe, from collected data. The method may also calculate calories andrecommend changes to the generated recipe based on the user's health ordiet plan. The method can improve its precision of recipe generationbased on the feedback received by the user.

The recipe generation can be fully automated with help of an intelligentidentification system. The method can determine the way of cooking,duration of cooking, step of cooking, the volume of ingredients, thequantity of salt, sugar and oil based on measured data. The system mayalso calculate calories and recommend changes to the generated recipebased on a user's health or diet plan.

The embodiments described below disclose a method to identify a preparedfood by taking a visual image, multispectral information, depthinformation and/or mass spectrography information using a camera. Themeasured data is analyzed to identify a meal and the ingredients of ameal. The proposed disclosure also provides a system including a botagent that iteratively provides feedback associated with capturing theimages to a user for precisely identifying a meal (for example, cut themeal into halves and take an image of an inside of the meal). The systemgenerates a recipe upon identifying the meal. The system may alsocalculate calories and recommend changes to the generated recipe basedon the user's health or diet plan.

FIG. 1 schematically describes an embodiment of a system for recipegeneration. The system for recipe generation is composed of a sensorarrangement 14, a processor 15, and a user interface 13.

The sensor arrangement 14, according to this example, is supposed to beon a mobile device such as a smart phone, a tablet or a wearable device.The processor 15 is also located on a mobile device. However, accordingto other embodiments, the processor 15 may also be located in part or asa whole on a remote workstation or cloud server. The user interface 13may be associated with visual feedback, e.g. on a mobile device or thevoice feedback via a mobile device.

FIG. 2 describes an example of a recipe generation process. At 201, datafrom a meal is acquired (e.g. by means of sensor arrangement 14 in FIG.1 ). The data, according to this embodiment, is comprised ofmultispectral images captured by a multispectral sensor, but it can bevarious kinds of information associated with various devices, such asvisible images, infrared images, spectral data and depth information. At202, information about the meal, e.g. the meal content, is determined byanalyzing measured data by utilizing computer vision techniques. Forinstance, based on given multispectral images, ingredient segmentationis employed by distinguishing the difference of spectrum properties ofingredients. Also, ingredients can be identified by analyzing spectrumdata. Depth information may be used for volume analysis of a whole mealor each ingredient to determine the quantity of the ingredients forrecipes. Mass spectroscopy data may be utilized to achieve precisecontent determination. This allows identifying various kinds ofcompounds including sugars, salts, oil, and biomolecules such asproteins. Conventional image data can be used for course identificationsuch as the meal name identification. This may help improving theefficiency of processing by narrowing down the candidate meal contents.

Finally, at 203, a recipe of the meal is generated. This can be done bycomparing and matching the meal content of an identified meal with thoseof meals on an existing recipe database, or more sophisticatedly, by afull automatic recipe generation process. From meal contentdetermination to recipe generation, the method may for example identifythe meal name by utilizing machine leaning and refer to an existingrecipe database based on the identified meal name to retrieve the recipeof identified meal from the recipe database.

FIG. 3 shows an embodiment of a recipe generation process associatedwith user feedback. As in the example of FIG. 2 , at 301, data from ameal is acquired, e.g. by means of sensor arrangement 14 in FIG. 1 . At302, information about the meal, e.g. the meal content, is determined byanalyzing measured data by utilizing computer vision techniques. At 303,it is determined if there is sufficient data for generating a recipe ofthe meal. If it is decided at 303 that there is sufficient data forgenerating a recipe of the meal, the process continues at 304. At 304, arecipe is generated for the meal based on the determined meal content.If it is decided at 303 that there is insufficient data for generating arecipe of the meal, the process continues at 305. At 305, the processgenerates a query to guide a user to provide more information on themeal. This process may be repeated iteratively until the whole mealcontent has been determined or until the identification of the meal canbe performed. At each step, feedback can be received from a user toachieve more precise recipe generation. For example, if the processcannot decide the meal content since the available data is notsufficient, or the data includes some ambiguities, the system caniteratively guide a user to check the meal to identify the contentprecisely.

It should be noted that not necessarily the whole meal content must bedetermined to identify a meal. According to some embodiments, thecontent of a meal is identified up to a predetermined level at which anidentification of the meal can be performed. If the meal can be fullyidentified based on only parts of its content, then a reference databasemay be queried to receive the remaining content of the meal that havenot yet been identified. It also should be noted that the user mayselect a meal from the identified candidates displayed on a display ofthe mobile device. The selection may be used to update the database ofthe recipes and to improve the machine learning.

The guidance may be generated according to the insufficient data.Exemplifying relationship between insufficient data and respectiveguidance is given in the embodiments described below with reference toFIGS. 4 to 11 .

FIG. 4 shows a specific embodiment of a recipe generation processassociated with user feedback. At 401, data from a meal is acquired,e.g. by a multispectral camera that is integrated in a mobile phone. At402, information about the meal, e.g. the meal content, is determined byanalyzing measured data by utilizing computer vision techniques. At 403,it is determined if ingredients overlap each other so that the cameracannot see all from one view angle. If it is decided at 403 thatingredients do not overlap each other, the process continues at 404. At404, a recipe is generated for the meal based on the determined mealcontent. If it is decided at 403 that ingredients overlap each other,the process continues at 405. At 405, the process generates a query toguide a user to change the attitude of the camera to see the otheringredients (the rest of the meal). Changing the attitude may forexample comprise changing the position and/or orientation (shootingangle) of the camera. This process may be repeated iteratively until thewhole meal content has been determined. This process may be particularlyhelpful if there is not enough spectral information available if theingredients overlap each other. In particular, it may allow to analyzeother ingredients of the meal.

FIG. 5 a shows an embodiment of a user query on a display of a mobilephone. The mobile phone 16 is equipped with a multispectral camera (e.g.on the back side of the device, thus not shown in FIG. 5 ) that isconfigured to analyze multispectral image data of a meal to obtaininformation on the contents of the meal. The mobile phone 16 includes adisplay screen 17 that acts as user interface.

The display screen 17 shows a query 18 to guide a user to change theattitude of the camera to point to red sauce that has been identifiedwithin a meal. A user having read this query 18 can react by changingthe attitude of the camera, e.g. the shooting angle of the camera (orthe angle of the mobile phone) to point to the red sauce. The processcan then repeat determining information about the meal by analyzingmeasured data, now based on the changed angle of the camera. This mayreveal better information on the meal, e.g. more specific nutritionfacts about the red sauce to which the camera is pointed may beobtained.

FIG. 5 b shows an alternative embodiment of a user query on a display ofa mobile phone. According to this embodiment, the mobile phone 16displays an image 19 of a meal as captured by the camera. Further, themobile phone 16 displays a query 18 that guides a user to change theattitude of the camera to point to the object in the meal as indicatedby an arrow 20 on the image 19. A user having read this query 18 canreact by changing the attitude of the camera, e.g. the shooting angle ofthe camera (or the angle of the mobile phone) to point to the objectidentified by the arrow 20. The process can then repeat determininginformation about the meal by analyzing measured data, now based on thechanged angle of the camera.

FIG. 6 shows a further specific embodiment of a recipe generationprocess associated with user feedback. At 601, data from a meal isacquired, e.g. by a multispectral camera that is integrated in a mobilephone. At 602, information about the meal, e.g. the meal content, isdetermined by analyzing measured data by utilizing computer visiontechniques. At 603, it is determined if an ingredient is insufficientlyshown (for instance, it being soaked in the soup). If it is decided at603 that all ingredients are sufficiently shown, the process continuesat 604. At 604, a recipe is generated for the meal based on thedetermined meal content. If it is decided at 603 that an ingredient isinsufficiently shown, the process continues at 605. At 605, the processgenerates a query to guide a user to pick up the ingredient and clearlyshow it to the camera. This process may be repeated iteratively untilthe whole meal content has been determined. This process may beparticularly helpful if there is not enough spectral informationavailable if the ingredients are not clearly shown to the camera. Inparticular, it may allow to adequately acquire images or data of themeal.

FIG. 7 a shows an embodiment of a user query on a display of a mobilephone. The mobile phone 16 is equipped with a multispectral camera (e.g.on the back side of the dive, thus not shown in FIG. 5 ) that isconfigured to analyze multispectral image data of a meal to obtaininformation on the contents of the meal. The mobile phone 16 includes adisplay screen 17 that acts as user interface. The display screen 17shows a query 18 to guide a user to pick up a pancake that has beenidentified in the meal and clearly show it to the camera. A user havingread this query 18 can react by picking up a pancake and showing it tothe camera. The process can then repeat determining information aboutthe meal by analyzing measured data, now based on the changed attitudeof the camera. This may reveal better information on the meal, e.g. morespecific nutrition facts about the pancake that is shown to the cameramay be obtained.

FIG. 7 b shows an alternative embodiment of a user query on a display ofa mobile phone. According to this embodiment, the mobile phone 16displays an image 19 of a meal as captured by the camera. Further, themobile phone 16 displays a query 18 that guides a user to pick up aspecific part of the meal as indicated by an arrow 20 on the image 19and clearly show it to the camera. A user having read this query 18 canreact by picking up a specific part of the meal as indicated by arrow20. The process can then repeat determining information about the mealby analyzing measured data, now based on better image data obtained fromthe specific part of the meal as indicated by an arrow 20.

FIG. 8 shows a further specific embodiment of a recipe generationprocess associated with user feedback. At 801, data from a meal isacquired, e.g. by a multispectral camera that is integrated in a mobilephone. At 802, information about the meal, e.g. the meal content, isdetermined by analyzing measured data by utilizing computer visiontechniques. At 803, it is determined if an ingredient is insufficientlyshown (for instance, it being covered with sauce). If it is decided at803 that all ingredients are sufficiently shown, the process continuesat 804. At 804, a recipe is generated for the meal based on thedetermined meal content. If it is decided at 803 that an ingredient isinsufficiently shown, the process continues at 805. At 805, the processgenerates a query to guide a user to cut the meal in half and show thesurface profile towards the camera. This process may be repeatediteratively until the whole meal content has been determined. Thisprocess may be particularly helpful if there is not enough spectralinformation available if a meal is not cut into parts. In particular, itmay allow to acquire images or data of a surface profile of the meal.

FIG. 9 shows an embodiment of a user query on a display of a mobilephone. The mobile phone 16 is equipped with a multispectral camera (e.g.on the back side of the dive, thus not shown in FIG. 5 ) that isconfigured to analyze multispectral image data of a meal to obtaininformation on the contents of the meal. The mobile phone 16 includes adisplay screen 17 that acts as user interface. The display screen 17shows a query 18 to guide a user to cut the meal into parts and show asurface profile of the meal towards the camera. A user having read thisquery 18 can react by cutting the meal into parts and by showing asurface profile of the meal towards the camera. The process can thenrepeat determining information about the meal by analyzing measureddata, now based on the information obtained from the surface profile ofthe meal. This may reveal better information on the meal, e.g. morespecific nutrition facts about the meal that is cut into parts may beobtained.

FIG. 10 shows a further specific embodiment of a recipe generationprocess associated with user feedback. At 1001, data from a meal isacquired, e.g. by a multispectral camera that is integrated in a mobilephone. At 1002, information about the meal, e.g. the meal content, isdetermined by analyzing measured data by utilizing computer visiontechniques. At 1003, it is determined if lots of ingredients are on aplate and it is too much information from wide angle view. If it isdecided at 1003 that that there are not too many ingredients on theplate, the process continues at 1004. At 1004, a recipe is generated forthe meal based on the determined meal content. If it is decided at 1003that there are too many ingredients on the plate, the process continuesat 1005. At 1005, the process generates a query to guide a user to movethe camera and to see a particular object close up. This process may berepeated iteratively until the whole meal content has been determined oruntil the identification of the meal can be performed. This process maybe particularly helpful if there is not enough depth informationavailable from a wide angle view. In particular, it may allow to executemore accurate volume analysis by excluding ambiguousness due to too muchinformation.

It should again be noted that not necessarily the whole meal contentmust be determined to identify a meal. According to some embodiments,the content of a meal is identified up to a predetermined level at whichan identification of the meal can be performed. If the meal can be fullyidentified based on only parts of its content, then a reference databasemay be queried to receive the remaining content of the meal that havenot yet been identified, respectively the recipe for cooking thecomplete meal.

FIG. 11 shows an embodiment of a user query on a display of a mobilephone. The mobile phone 16 is equipped with a multispectral camera (e.g.on the back side of the dive, thus not shown in FIG. 5 ) that isconfigured to analyze multispectral image data of a meal to obtaininformation on the contents of the meal. The mobile phone 16 includes adisplay screen 17 that acts as user interface. The display screen 17shows a query 18 to guide a user to move the camera to see a pancake inthe meal close up. A user having read this query 18 can react by movingthe camera to see the pancake in the meal close up. The process can thenrepeat determining information about the meal by analyzing measureddata, now based on the information obtained from the surface profile ofthe meal. This may reveal better information on the pancake, e.g. morespecific nutrition facts about the pancake that is examined close up maybe obtained.

The processes of FIGS. 6, 8 and 10 may be combined and executediteratively (e.g. one after another and then repeated) to analyze a mealcompletely. If all ingredients of a meal are identified, a recipe of themeal can be generated.

FIGS. 12 a, b, c show an example of a generated recipe.

FIG. 12 a shows a recipe as it may be displayed on the screen of amobile device. The recipe includes three subsections, “Ingredients” 21,“Nutrition information” 22 and “Allergen information” 23. In FIG. 12 a ,the “Ingredients” section 21 is selected and displayed. At 24, the nameof the identified meal is displayed, here “Pancakes with cheese andtomato sauce”. At 25, the ingredients of the meal are identified. In theembodiment, a pancake with the ingredients wheat flour, water, yeast,sugar, vegetable oil, iodized salt, gluten, soy and flour has beenidentified. Still further, cheese with the ingredients milk, salt,culture, enzymes, water, milk solids, butter, emulsifiers, salt, acidityregulators, colors, and soy lecithin has been identified. Still further,a sauce with the ingredients water, tomato paste, high fructose cornsyrup, vinegar, and salt has been identified.

In FIG. 12 b , the “Nutrition information” section 22 is selected anddisplayed. At 26, the nutrition information of the meal is identified.In the embodiment, the meal is identified as including 1200 kJ energy(which is 280 Cal), 12.2 g protein, 11.8 g fat, 28.1 g Carbohydrate, 5.7g sugars and 572 mg sodium.

In FIG. 12 c , the “Allergen Information” section 23 is selected anddisplayed. At 27, the allergen information of the meal is identified. Inthe embodiment, the meal is identified as including gluten, traces ofegg, milk, soy and flavors as potential allergens.

According to still further embodiments, the recipe generation alsoincludes determining cooking instructions that result in a meal thatcorresponds to the result (ingredients, nutrition information) of theabove described process of computer vision. That is, based on theingredients and nutrition information of a meal, a recipe of the mealcan be obtained. Not only the meal name can be identified, but also themeal content and cooking operations can be determined. A recommendationapplication may recommend a recipe according to a build model of auser's typical food preferences and choices by analyzing the food intakeover time, or it may provide recommendations based on a person'spurchase history stored in a database, or it may recommend a recipeaccording to user's diet plan.

The system may for example provide a web link to a web page providingthe steps of cooking.

The recipe generation can be fully automated with the help of theabove-described intelligent identification system. The method candetermine the way of cooking, duration of cooking, step of cooking, thevolume of ingredients, the quantity of salt, sugar and oil based onmeasured data. The system may also calculate calories and recommendchanges to the generated recipe based on the user's health or diet plan.

FIG. 13 schematically describes an embodiment of a system for recipegeneration via Internet. The system generates recipes 37 by visualobservation. Multispectral images 31 of a meal 32 are captured by amultispectral imaging device 33. The measured data 31 is transferred tocomputational unit 34 via Internet 35. On the computational unit 34, arecipe 37 is determined by machine learning based on multispectralimages 31 referring to existing recipe databases 36. The recipe 37 canbe composed of several recipe candidates shown to a user.

FIG. 14 schematically describes an embodiment of a system for recipegeneration via Internet using feedback. The system generates a recipe 37with interactive feedback from a user. Multispectral images 31 of a mealare captured by multispectral imaging device 33. The measured data 31 istransferred to computational unit 34 via Internet 35. At thecomputational unit 34, it is determined that the information is notenough to determine a recipe. Accordingly, the system sends a query 38with some guidance for asking a user to change the capturing setting(i.e. to capture the meal differently). The query 38 can for example bean advice for changing the attitude of camera, or an advice for cuttingthe meal in half, or an advice for closing up particular ingredient.After enough information to identify ingredients has been acquired, thesystem generates a recipe 37 and sends it to a user.

FIG. 15 shows an embodiment of a method for recipe generation in moredetail. According to this embodiment, a spectral imager (e.g. spectralsensor) and a calibrated light source is utilized. By illuminating ameal first without calibrated light source (that is, by e.g. ambientlight) a spectral image 601 is taken. Afterwards, a second spectralimage 602 is taken with calibrated light source. At 603, a spectralimage 604 is generated by subtraction which corresponds to the conditionif only light of the calibrated light source is present. From this, at605, a reflectance image is calculated. From this reflectance image, at606, by preprocessing steps reflectance features (like ratios ordifferences between different wavelengths) are derived. Simultaneously,at 607, an RGB image is derived from the spectral image 604. From thisRGB image, at 608, meal parts if present are analyzed and respectivefeatures like the size of dumplings are derived. At 609, based on thecombined features, reflectance and feature analysis, the meal analysisis accomplished. This process might take into account an additionaldatabase 610 containing further information about the identified orclassified meals and its components. At 611, the results of the mealanalysis is output as a recipe.

It should be noted that the creation of an RGB as done at 607 in theprocess of FIG. 6 is not necessarily required as the spectral imageitself could be used directly for a feature analysis.

FIG. 16 shows another embodiment of a method for recipe generation.According to this embodiment, a miniaturized spectrometer, RGB imagerand a calibrated light source is utilized. By illuminating a meal firstwithout calibrated light source a spectral image 701 is taken.Afterwards, a second spectral image 702 is taken with calibrated lightsource. At 703, a spectral image 704 is generated by subtraction whichcorresponds to the condition if only light of the calibrated lightsource is present. From this spectral image 704, at 705, its reflectanceis obtained. From this reflectance, at 706, by preprocessing steps,reflectance features are derived. Simultaneously, at 712, an RGB imageis taken with RGB imager under ambient light conditions. As in theembodiment of FIG. 15 , the RGB image allows, at 713, analyzing mealparts if present and deriving respective features. Based on the combinedfeatures, reflectance and feature analysis, at 709, the meal analysis isaccomplished. Again, this step might take into account an additionaldatabase 710 containing further information about identified orclassified meals and its components. At 711, the results of the mealanalysis are output as a recipe.

In the following, the measurement of the reflectance of a liquidaccording to an embodiment is described. In this embodiment, themeasurement of the reflectance for the object, here a liquid, is done byperforming two spectral measurements, wherein FIG. 17 a illustrates afirst measurement and FIG. 17 b illustrates a second measurement.

A first measurement takes place (FIG. 17 a ), when the light source isswitched off, e.g. by the processor 5, such that only light from anambient light source 7, such as the sun or other light source, ispresent.

Then, the processor drives the spectral sensor 4 accordingly to collectfirst spectral information about light which is reflected by the object8 in the form of a spectral image or a spectrum S_(A) and whichincidents into the spectral sensor 4. The spectrum S_(A) can be storedin a memory, storage or the like of the reflectometer 1.

For the second measurement, the calibrated light source 2 is switchedon, e.g. by the processor 5. Now, ambient light emitted from the ambientlight source 7 and light from the calibrated light source 2 illuminatethe object of interest. The spectral sensor 4 collects second spectralinformation in the form of a spectral image or a spectrum S_(A)+F forthe light reflected from the object 8 originating from the calibratedlight source 2 and the ambient light source 7. Hence, the reflectedlight includes light from the ambient light source 7 and light from thecalibrated light source 2.

Additionally, at the same time of the second measurement and the sametime as the spectral sensor 4 is driven by the processor 5, theprocessor 5 also drives the depth sensor 3, which determines a distancebetween the depth sensor 3 and the object 8 by capturing a depth map D.It is assumed that the relative distance between object 8 andreflectometer 1 is the same in both measurements. Of course, the pointof time of driving the depth sensor 3 is only exemplary, and, inprinciple, the depth sensor 3 can be driven at any point of time forobtaining the depth map D.

The spectra S_(A) and S_(A+F), the depth map D and other parameters maybe stored by the processor 5 in a memory, storage or the like.

After having performed the two measurements, the processor 5 calculatesthe absolute reflectance spectrum as follows and as also illustrated inFIG. 18 :

First, a spectrum S_(F) is calculated, which represents lightintensities reflected from the object 8 as if only light were reflectedfrom the object 8 originating from the light source 2. This is done bysubtracting the spectrum S_(A+F) obtained during the second measurementwhere the light source 2 was switched on and the spectrum S_(A) obtainedin the first measurement where the light source 2 was switched off fromeach other:S _(F) =S _(A+F) −S _(A)

Second, the absolute power IF of the calibrated light source 2 at theposition of the object 8 is calculated by the processor 5.

In the coordinate system of the depth sensor 3 the object 8 is locatedat (d_(D), r_(D), (φ_(D)), see also FIG. 18 .

This can be directly derived from the acquired depth map D whichincludes the information of the distance of the object 8 with respect tothe depth sensor 4.

The processor 5 performs a simple coordinate transformation T, whichresults in the coordinates (d_(F), r_(F), φ_(F)) in the coordinatesystem of the calibrated light source 2:(d _(F) ,r _(F),φ_(F))^(T) =T*(d _(D) ,r _(D),φ_(D))^(T)

These coordinates (d_(F), r_(F), φ_(F)) can be used for calculating theabsolute incident power I_(F), as introduced before:I _(F) =I(d _(F) ,r _(F),φ_(F)).

Finally, the absolute reflectance R is obtained by dividing thereflected power S_(F) with the incident power IF:R=S _(F) /I _(F)

As mentioned above, in the present embodiment the depth sensor 3 and thespectral sensor 4 are very close to each other such that the influenceof the distance between them is negligible. In other embodiments, thedistance between the depth sensor 3 and the spectral sensor 4 can beconsidered by performing another coordinate transformation, for example,into the coordinate system of the spectral sensor 4. However, then theclassical parallax problems, such as occlusion, may arise.

In the present embodiment, the calculation was done for a single pointof an object. In other embodiments, the depth sensor and/or the spectralsensor may be two-dimensional (2D) sensors such that also a complete 2Dreflectance measure may be performed in such embodiments. Moreover, thesingle point measurement as done in the embodiment discussed above canalso be repeated for multiple points of an object.

All units and entities described in this specification and claimed inthe appended claims can, if not stated otherwise, be implemented asintegrated circuit logic, for example on a chip, and functionalityprovided by such units and entities can, if not stated otherwise, beimplemented by software.

The methods as described herein are also implemented in some embodimentsas a computer program causing a computer and/or a processor to performthe method, when being carried out on the computer and/or processor. Insome embodiments, also a non-transitory computer-readable recordingmedium is provided that stores therein a computer program product,which, when executed by a processor, such as the processor describedabove, causes the methods described herein to be performed.

It should be recognized that the embodiments describe methods with anexemplary order of method steps. The specific order of method steps is,however, given for illustrative purposes only and should not beconstrued as binding.

The method can also be implemented as a computer program causing acomputer and/or a processor to perform the method, when being carriedout on the computer and/or processor. In some embodiments, also anon-transitory computer-readable recording medium is provided thatstores therein a computer program product, which, when executed by aprocessor, such as the processor described above, causes the methoddescribed to be performed.

In so far as the embodiments of the disclosure described above areimplemented, at least in part, using a software-controlled dataprocessing system, it will be appreciated that a computer programproviding such software control and a transmission, storage or othermedium by which such a computer program is provided are envisaged asaspects of the present disclosure.

Note that the present technology can also be configured as describedbelow.

(1) A system including

-   -   circuitry configured to    -   process multispectral image data of a meal to obtain information        on the contents of the meal;    -   generate, based on the obtained information, a query with        guidance to change image capture settings.

(2) The system of (1), wherein the circuitry is configured to generatethe query with guidance according to insufficient information on thecontents of the meal.

(3) The system of (1) or (2), wherein the circuitry is configured toguide a user to change a shooting angle of a camera to point to otheringredients of the meal.

(4) The system of anyone of (1) to (3), wherein the circuitry isconfigured to guide the user to pick up at least a part of the meal.

(5) The system of anyone of (1) to (4), wherein the circuitry isconfigured to guide the user to cut the meal into parts and show asurface profile of the meal towards a camera.

(6) The system of anyone of (1) to (5), wherein the circuitry isconfigured to guide the user to move a camera and to see a particularobject in the meal close up.

(7) The system of anyone of (1) to (6), wherein the circuitry isconfigured to generate a recipe of the meal based on the obtainedinformation on the contents of the meal.

(8) The system of (7), wherein the recipe includes ingredientsinformation, nutrition information, and/or allergen information.

(9) The system of (7) or (8), wherein the circuitry is configured tochange the recipe generation process based on feedback.

(10) The system of anyone of (1) to (9), further including a sensorarrangement configured to collect multispectral image data of a meal.

(11) The system of (10), wherein the sensor arrangement is configured toprovide depth information.

(12) The system of (10) or (11), wherein the sensor arrangement isconfigured to provide mass spectrography information.

(13) The system of anyone of (10) to (12), wherein the sensorarrangement is configured to provide visible images, infrared images,and/or spectral data.

(14) The system of anyone of (1) to (13), wherein the circuitry isconfigured to perform ingredient segmentation on a multispectral imageby distinguishing the difference of spectrum properties of ingredients.

(15) The system of anyone of (1) to (14), wherein the circuitry isconfigured to identify ingredients by analyzing spectrum data.

(16) The system of anyone of (1) to (15), wherein the circuitry isconfigured to use conventional image data for course identification.

(17) A method including

-   -   processing multispectral image data of a meal to obtain        information on the contents of the meal; and    -   generating, based on the obtained information, a query with        guidance to change image capture settings.

(18) A computer program including instructions, the instructions whenexecuted on a processor causing the processor to perform:

-   -   processing multispectral image data of a meal to obtain        information on the contents of the meal; and    -   generating, based on the obtained information, a query with        guidance to change image capture settings.

(19) A non-transitory computer-readable medium embedded with a program,which when executed by a computer, causes the computer to perform amethod including:

-   -   processing multispectral image data of a meal to obtain        information on the contents of the meal; and        generating, based on the obtained information, a query with        guidance to change image capture settings.

The invention claimed is:
 1. A system comprising: circuitry configuredto process image data of a meal to obtain information on the contents ofthe meal; generate, based on the obtained information, a query withguidance to change image capture settings; and guide a user to pick upat least a part of the meal.
 2. The system of claim 1, wherein thecircuitry is configured to generate the query with guidance according toinsufficient information on the contents of the meal.
 3. The system ofclaim 1, wherein the circuitry is configured to guide the user to changean attitude of a camera to point to other ingredients of the meal. 4.The system of claim 1, wherein the circuitry is configured to guide theuser to cut the meal into parts and show a surface profile of the mealtowards a camera.
 5. The system of claim 1, wherein the circuitry isconfigured to guide the user to move a camera and to see a particularobject in the meal close up.
 6. The system of claim 1, wherein thecircuitry is configured to generate a recipe of the meal based on theobtained information on the contents of the meal.
 7. The system of claim6, wherein the recipe includes ingredients information, nutritioninformation, and/or allergen information.
 8. The system of claim 6,wherein the circuitry is configured to change the recipe generationprocess based on feedback.
 9. The system of claim 1, further comprisinga sensor arrangement configured to collect the image data of a meal. 10.The system of claim 9, wherein the sensor arrangement is configured toprovide depth information.
 11. The system of claim 9, wherein the sensorarrangement is configured to provide mass spectrography information. 12.The system of claim 9 wherein the sensor arrangement is configured toprovide visible images, infrared images, and/or spectral data.
 13. Thesystem of claim 1, wherein the circuitry is configured to performingredient segmentation on a image by distinguishing the difference ofspectrum properties of ingredients.
 14. The system of claim 1, whereinthe circuitry is configured to identify ingredients by analyzingspectrum data.
 15. The system of claim 1, wherein the circuitry isconfigured to use conventional image data for course identification. 16.A method comprising: processing image data of a meal to obtaininformation on the contents of the meal; generating, based on theobtained information, a query with guidance to change image capturesettings; and guiding the user to pick up at least a part of the meal.17. A non-transitory computer-readable recording medium that stores aprogram which causes a computer to execute a method, the methodcomprising: processing image data of a meal to obtain information on thecontents of the meal; generating, based on the obtained information, aquery with guidance to change image capture settings; and guiding theuser to pick up at least a part of the meal.